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Abstract 

Background: Improving the coordination of cancer care is a priority area for service improvement. However, 
quality improvement initiatives are hindered by the lack of accurate and reliable measures of this aspect of cancer 
care. This study was conducted to develop a questionnaire to measures patients' experience of cancer care 
coordination and to assess the psychometric properties of this instrument. 

Methods: Questionnaire items were developed on the basis of literature review and qualitative research involving 
focus groups and interviews with cancer patients, carers and clinicians. The draft instrument was completed 686 
patients who had been recently treated for a newly diagnosed cancer, including patients from metropolitan, 
regional and rural areas of New South Wales, Australia. To assess test-retest reliability, 119 patients completed the 
questionnaire twice. Unreliable items those with limited variability or high levels of missing data were eliminated. 
Exploratory factor analysis was conducted to define the underlying factor structure of the remaining items and 
subscales were constructed. Correlations between these and global measures of the experience of care 
coordination and the quality of care were assessed. 

Results: Of 40 items included in the draft questionnaire, 20 were eliminated due to poor test-retest reliability 
(n = 4), limited response distributions (n = 8), failure to load onto a factor (n = 7) or detrimental effect on the 
internal consistency of the scale (n = 1). The remaining 20 items loaded onto two factors named 
'Communication' and 'Navigation', which explained 91% of the common variance. Internal consistency was 
with high for the instrument (Cronbach's alpha 0.88) and each subscale (Cronbach's alpha 0.87 and 0.73 
respectively). There was no apparent 'floor' or 'ceiling' effect for the total score or the Communication 
subscale, but evidence of a ceiling effect for the Navigation subscale with 21% of respondents achieving the 
highest possible score. There were moderate positive associations between the total score and global 
measures of care coordination (r = 0.57) and quality of care (r = 0.53). 

Conclusions: The instrument developed in this study demonstrated consistency and robust psychometric 
properties. It may provide a useful tool to measure patients' experience of cancer care coordination in future 
surveys and intervention studies. 
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Background 

Effective coordination of care between different clini- 
cians, services and health sectors throughout the patient 
journey is fundamental to the provision of high-quality 
care [1-3]. In health systems where care is well coordi- 
nated, patients will experience effective flow of informa- 
tion between clinicians throughout the course of their 
illness, with streamlined service provision in response to 
their physical, emotional and social needs [4]. Not only 
is good care co-ordination essential to optimize patients' 
experience, but it has also been shown to reduce future 
need for supportive care and to improve psychosocial 
outcomes [5]. 

People with cancer are particularly at risk of receiving 
poorly organized and fragmented care due to the com- 
plex nature of the disease and its management, which 
often involves multidisciplinary care from a large team 
of medical, nursing and allied health practitioners in 
both hospital and community settings over extended 
periods of time. As a result, many national strategic can- 
cer plans have identified the improvement of cancer 
care coordination as a priority for service improvement 
[1,4,6,7]. 

Efforts to improve cancer care coordination to date 
have been hindered by a dearth of accurate and reliable 
measures by which progress can be monitored. This 
partly stems from the lack of an agreed theoretical frame- 
work or definition of the term 'care coordination' to 
underpin the development of measures. For example, a 
recent literature review prepared for the Agency for 
Healthcare Research and Quality (AHRQ) identified 
more than 40 different definitions for 'care coordination'. 
However, the authors identified a number of common 
elements to inform the following working definition: 

'Care coordination is the deliberate organization of 
patient care activities between two or more partici- 
pants (including the patient) involved in a patient's 
care to facilitate the appropriate delivery of health 
care services. Organizing care involves the marshaling 
of personnel and other resources needed to carry out 
all required patient care activities, and is often man- 
aged by the exchange of information among partici- 
pants responsible for different aspects of care [8]. 

This definition provides a starting point to identify the 
specific aspects of the care experience that should be 
addressed in any measurement tool. 

There are a number of sources of data that could be 
used to assess aspects of cancer care coordination, 
including administrative health datasets, audits of indivi- 
dual patient records or measures based on the experi- 
ence of patients or clinicians. Patients are ideally placed 



to rate the adequacy of cancer care coordination, how- 
ever, as they are likely to be the only individual present 
at every encounter with health services. Furthermore, 
the move towards more patient-centred care in which 
services are organized around the needs and preferences 
of individual patients, emphasizes the primacy of mea- 
sures based on patients' own experience. Therefore we 
conducted this study to develop a questionnaire for 
patients to assess their experience of cancer care coordi- 
nation in the treatment phase of the cancer journey, to 
define the underlying factor structure of the question- 
naire and to conduct initial validation by assessing con- 
struct validity, internal consistency and test-retest 
reliability. 

Methods 

Item generation and development of a draft 
questionnaire 

A literature review was undertaken to identify relevant 
issues and terminology as well as items and scales 
within existing instruments that could be used to mea- 
sure aspects of cancer care coordination [8-13]. The lit- 
erature review was used to develop a series of open- 
ended questions that were used in a qualitative study to 
explore issues in care coordination specific to oncology. 
Focus groups and semi-structured interviews with 24 
patients and carers and 29 clinicians in metropolitan, 
regional and rural areas of New South Wales (NSW) 
were undertaken to investigate stakeholders' views of 
the most important components of cancer care coordi- 
nation and to identify potential questionnaire items. Full 
details of this qualitative study are reported elsewhere 
[14] but in brief, eight components of care were identi- 
fied as being crucial for effective cancer care coordina- 
tion, namely, organisation of patient care, access to and 
navigation through the healthcare system, the allocation 
of a "key contact" person, recognition and understand- 
ing of medical team roles, effective communication and 
cooperation amongst the multidisciplinary team and 
other health service providers, delivery of services in a 
complementary and timely manner, needs assessment 
and sufficient and timely information for the patient. 

The results of the literature review and this qualitative 
work were used to identify existing items and to gener- 
ate new items that addressed these eight components of 
cancer care coordination as well as the concepts 
espoused in the AHRQ definition [8]. To generate new 
items, the study team developed statements that 
addressed the concept in question and sought input 
from clinicians and other researchers about clarity and 
wording. 

Forty items that related to concepts considered impor- 
tant by a broad range of stakeholders in the qualitative 
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phase and that addressed the theoretical components of 
cancer care coordination were selected for inclusion in a 
draft questionnaire. Items were worded both in the posi- 
tive and the negative with holding of words used to 
highlight differences between similar statements. To 
investigate the most reliable format for response 
options, two formats were tested. Eighteen items were 
phrased as statements to which respondents were asked 
to indicate their level of agreement, using a five-point 
Likert scale {'Strongly agree', 'Agree', 'Neutral', 'Dis- 
agree', 'Strongly disagree'). The remaining 22 items 
asked about patients' experiences of care in the previous 
three months, again using a five-point Likert scale 
('Never', 'Rarely', 'Sometimes', 'Frequently', 'Always'). A 
time frame of three months was chosen on clinical 
grounds to provide a sufficient time window for patients 
to have received multidisciplinary cancer care. The 
items with the 'agreement' format were included in ran- 
dom order, followed by the items using the 'experience' 
format, again in random order. The response option 
headings were repeated at intervals down the page to 
break up the lines of text and tick boxes so as to 
improve the ease of completing the questionnaire. In 
addition, the questionnaire included two global assess- 
ment questions in which respondents were asked to rate 
firstly, the coordination of their care and secondly, the 
overall quality of the care they had received, on a scale 
from one ('Very poor') to ten ('Excellent'). The draft 
questionnaire was reviewed by clinicians and researchers 
to assess comprehensiveness of items (face validity) and 
clarity of wording. 

The draft questionnaire was then tested in two sepa- 
rate samples of patients. 

Sample 1 

A purposive sample was recruited from six centres (two 
in Sydney, four in regional New South Wales (NSW)) to 
provide patients with a range of cancer types, treatment 
modalities and geographical location. Eligible patients 
were in follow up for any cancer that had been treated 
between three and twelve months previously. This time- 
frame was considered optimal as patients would have 
experienced the full range of care-co-ordination through 
the treatment phase of their illness. Patients were con- 
sidered ineligible if they had insufficient English skills or 
were cognitively impaired such that they could not com- 
plete the questionnaire or were receiving end of life 
care. 

Patients were asked to read and sign a consent form, 
complete the questionnaire and return these items to 
the research team in a reply paid envelope. In addition, 
patients completed items assessing demographic and 
clinical information, including age; sex; country of birth; 
marital, education and occupational status; cancer type, 



year of diagnosis and treatment modalities. To assess 
test-retest reliability, on receipt of their completed ques- 
tionnaire, patients in the first three month period of 
recruitment were mailed a second, identical copy of the 
questionnaire to complete two weeks later. 

Sample 2 

This sample comprised patients with a newly diagnosed 
colorectal cancer who were participating in an ongoing 
randomised trial. Patients treated at 22 public and pri- 
vate hospitals in metropolitan and regional centres in 
NSW were recruited at the time of initial surgical treat- 
ment and asked to complete self-administered question- 
naires at baseline, one, three and six months. The data 
for the present study are from the 3 -month assessment 
which included the draft questionnaire about cancer 
care coordination. Demographic and clinical information 
was collected at the time of enrolment into the trial. 

Statistical analysis 

Characteristics of participants were summarized. For the 
subsample of Sample 1 who completed the question- 
naire twice, test-retest reliability (repeatability) of indivi- 
dual items was assessed by calculating weighted Kappa 
statistics with 95% confidence intervals (CIs). Items with 
kappa values of less than 0.40, representing 'fair' or 
'poor' agreement, [15] were eliminated from further 
analyses. 
Item reduction 

Using the combined dataset (n = 686), frequency distri- 
butions for each item were examined. Items with more 
than 5% missing data and those with limited response 
distributions (70% or more respondents gave the same 
response) were eliminated [16]. 
Investigation of factor structure 

Study data were then randomly split into two equal 
sized sub-samples. Exploratory factor analysis using 
squared multiple correlations as prior communality esti- 
mates was conducted in each sub-sample separately to 
assess the consistency of the factor pattern. The princi- 
pal factor method was used to extract the factors, fol- 
lowed by a promax rotation [17,18]. The number of 
meaningful factors was determined on the basis of 
examination of the scree plot, assessment of the propor- 
tion of variance accounted for and interpretability of the 
factors. Factors that explained at least 5% of the com- 
mon variance were retained. For interpretation of the 
rotated factor pattern, an item was said to load onto a 
particular factor if the factor loading was greater than 
0.40 for that factor, and was less than 0.40 for the other 
factors. 

Development of subscales 

The factors were used to develop subscales within the 
questionnaire. First, the scoring for items worded in the 
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negative were reversed, so that a higher score indicated 
better care coordination for all items. Items that loaded 
onto each factor were summed to create factor-based 
scales. To assess whether any individual items reduced 
the internal consistency of the total score or individual 
subscales, item-total correlations were calculated. These 
statistics provide a measure of the correlation between 
an item and the sum of the remaining items in the 
scale, with low values (less than 0.2) [16] indicating that 
an item is not measuring the same construct as other 
items. Cronbach's alpha was calculated with each item 
removed in turn. Where Cronbach's alpha was substan- 
tially improved by removal of the item, this item was 
eliminated from the scale and Cronbach's alpha for the 
remaining items was recalculated [16]. Values of Cron- 
bach's alpha between 0.7 and 0.9 were considered opti- 
mal [16]. Correlations between variables were assessed 
to determine whether any were highly correlated (r > 
0.70) suggesting redundancy. 

The distribution of subscale scores and the total score 
were assessed with descriptive statistics and the propor- 
tions of respondents with the highest ('ceiling') and low- 
est ('floor') scores were calculated. Spearman's rank 
correlation coefficient was calculated for the total score 
and each of the subscales firstly with the global cancer 
care coordination item and secondly with the global 
quality of care item. All statistical analyses were under- 
taken using SAS statistical software [19]. 

Sample size 

A sample of five times the number of questionnaire 
items is considered the minimum for factor analysis 
[20]. As the questionnaire contained 40 items, we 
needed a minimum of 200 patients in each split sample 
for this analysis. A minimum sample size of 50 is 
recommended for assessment of test-retest reliability 
[15]. 

Ethics approval 

The study was approved by the Sydney South West 
Area Health Service Ethics Review Committee (RPAH 
zone). 

Results 

Overall, 686 patients completed the questionnaire, 245 
from Sample 1 and 441 from sample 2. Characteristics 
of patients are summarized in Table 1. The mean age of 
participants was 66.1 years (sd 13.3 years), with a range 
of 23-98 years. 

Item reduction 

Among 119 patients in Sample 1 who completed the 
questionnaire twice, values of weighted Kappa for indivi- 
dual items ranged from 0.29 to 0.69. Four items with 



Table 1 Characteristics of respondents 





n 


% 


Sex 






Male 


365 


53.2 


Female 


321 


46.8 


Country of birth 






Australia 


521 


75.9 


Other 


165 


24.0 


Marital status 






Never married 


73 


10.6 


Married/living as married 


459 


66.9 


Other 


143 


20.8 


Education 






Primary school only or none 


68 


9.9 


Secondary school 


340 


49.6 


Tertiary degree or diploma 


242 


35.3 


Employment status 






Casual/part time/student 


60 


8.7 


Eull time 


161 


23.5 


Retired 


405 


59.0 


Unemployed 


47 


6.9 


Location 






Metropolitan 


409 


59.6 


Regiona 


277 


404 


Cancer Site 






Colorectal 


566 


82.5 


Gynaecologica 


52 


7.6 


Breast 


18 


2.6 


Lung/mesothelioma 


9 


1.3 


Other/multiple sites 


32 


47 


Cancer type 






Primary cancer 


630 


91.8 


Recurrent cancer 


26 


3.8 


Treatments received 






Surgery 


659 


96.0 


Chemotherapy 


278 


40.5 


Radiotherapy 


84 


12.2 


Hormone therapy 


27 


3.9 


^Where data are missing, categories do not sum to 100% 



values less than 0.40 were eliminated from further ana- 
lyses. Two of these used the 'agreement' response for- 
mat and two the 'experience' response format. Using the 
entire dataset (n = 686), eight items demonstrated lim- 
ited response distributions with 70% or more of the 
sample giving the same response and so were elimi- 
nated. These eight items each used the 'experience' 
response format. 

Construct validity 

The underlying constructs represented by the remaining 
items were explored in factor analysis using the split 
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sample approach. In the first sub-sample, a solution in 
which 21 items loaded onto two factors had a simple 
structure with all items having a high loading on one 
factor and a low loading on the other. Furthermore, 
each factor contained at least three items, considered 
the minimum for a robust subscale [19] and the items 
on each factor shared some conceptual meaning. When 
repeated using the second sub-sample, a consistent fac- 
tor pattern emerged and so results from an analysis 
using the entire dataset are reported. One item was 
eliminated subsequently as the internal consistency of 
the questionnaire improved substantially with its 
removal. One item was retained despite a factor loading 
of only 0.37 as this item had loaded more strongly in 
the separate subsamples and because it was the only 
one that addressed difficulties with access to general 
practitioners, an issue that had emerged from the quali- 
tative research as being of great concern, particularly for 
people living in regional and rural areas. The factors 
were labelled 'Communication' (13 items) and 'Naviga- 
tion' (7 items) (Table 2). The revised instrument demon- 
strated good internal consistency with values of 
Cronbach's alpha greater than 0.7 for both subscales 
and the total score (Table 3). Item to item correlations 



ranged from 0.19 to 0.65 for the Communication sub- 
scale and from 0.18 to 0.62 for the Navigation subscale. 

Although the Navigation subscale demonstrated a ceil- 
ing effect with 21% of the sample achieving the highest 
possible score, the Communication subscale and total 
score were both more Normally distributed with no evi- 
dence of ceiling or floor effects (Table 4). There were 
moderate positive correlations between the scales and 
the global measures of care coordination and quality of 
care (Table 3). The revised questionnaire is available in 
Additional File 1. 

Discussion 

Despite increasing recognition of inadequate care coor- 
dination as a common problem experienced by patients, 
to date there have been few measures by which 
improvement or deterioration in this crucial aspect of 
cancer care could be measured. The aim of this study 
was to develop a valid and reliable self-administered 
questionnaire for patients to measure the adequacy of 
cancer care coordination for those in the treatment 
phase of the cancer journey. The resulting questionnaire 
demonstrated robust psychometric properties and con- 
sistent subscales, suggesting that this instrument could 



Table 2 Factor structure and loadings 



Item 


Factor 1 


Factor 2 


1 knew the warning signs and symptoms 1 should watch for to monitor my health 


0.43 


0.14 


1 always knew what tests, treatments and follow up were planned for me 


0.70 


0.03 


1 knew whether chemotherapy or radiotherapy were suitable for me 


0.71 


0.11 


1 was fully informed about the advantages and disadvantages of any additional treatments (eg radiotherapy, chemotherapy 
or hormonal therapy) that were relevant to me 


0.75 


0.11 


1 always knew the reason why 1 was having a test or treatment 


0.64 


0.00 


1 had access to all the additional services (eg stoma therapy, counselling, cancer support groups, nutritional advice) that 1 
needed 


0.60 


0.07 


1 had sufficient help from staff with dealing with the emotional impact of my cancer 


0.60 


0.13 


1 had a good understanding of the things 1 was responsible for to help my treatment plan run smoothly 


0.65 


0.02 


1 had sufficient help from staff with practical arrangements 


0.67 


0.00 


1 was fully informed by staff about my financial entitlements (eg IVledicare and health fund claims, travel allowances etc) 


0.46 


0.17 


The health professionals looking after me always picked up on whether 1 was feeling anxious or down 


0.58 


0.11 


How often were you asked how your visits with other health professionals were going? 


0.46 


0.09 


How often were you asked how well you and your family were coping? 


0.50 


0.06 


How often were you unsure who you should contact if you had concerns about your health or treatment plan? 


0.06 


0.68 


How often were you unsure who to call out of business hours if you had a problem? 


0.03 


0.68 


How often were you confused about the roles of the different health professionals involved in your care? 


0.03 


0.67 


How often was it difficult to meet the financial costs associated with your health care? 


0.10 


0.45 


How often did you feel that health professionals looking after you were not fully informed about your history and 
progress? 


0.02 


0.52 


How often did you have difficulty getting an appointment with your GP? 


0.03 


0.37 


How often did you have to wait too long to get the first available appointment for a test or treatment? 


0.04 


0.43 


Eigenvalue 

% variance explained 


5.9 

74.3% 


1.4 

1 7.0% 
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Table 3 Internal consistency, Inter-Item correlations and correlations with other measures 





Communication 


Navigation 


Total score 


Cronbach's alpha 


0.87 


0.73 


0.88 


Correlation with global measure of care coordination 


0.50 


047 


0.57 


Correlation with global measure of quality of care 


048 


041 


0.53 



provide a useful tool to measure cancer care coordina- 
tion in future patient surveys and intervention studies. 

The process of developing a new questionnaire is 
lengthy, requiring a number of iterative steps to further 
refine the wording and items to provide an instrument 
that is both acceptable and easily-understood by the tar- 
get audience as well as providing a comprehensive, 
accurate and reliable measure of the phenomenon of 
interest. Furthermore, there is always a tension between 
the comprehensiveness of the instrument and the bur- 
den that a lengthy instrument will place on respondents. 
Brief instruments may achieve higher response rates, but 
may also limit the breadth or depth of information that 
can be collected. The approach to instrument develop- 
ment in this study was to only include items that had 
sound psychometric properties. Unreliable items that 
elicit inconsistent responses from an individual are of 
no value, as are items that are frequently missed out, 
perhaps due to lack of clarity in the wording or lack of 
relevance to a significant number of individuals. 
Furthermore, items that elicit highly skewed responses, 
with almost everyone giving a similar response, are of 
limited value for measurement. On the basis of these 
considerations, the draft questionnaire was reduced to 
20 items that demonstrated good internal consistency 
and addressed two important components of cancer 
care coordination, namely the issues of communication 
and navigation of the health care system. 

The Communication factor was the strongest, 
accounting for nearly 75% of the variance and demon- 
strating internal consistency in the desired range of 0.7- 
0.9 [16]. Comprising fewer items, the Navigation sub- 
scale necessarily had a lower value of Cronbach's alpha 
as this is partly dependent on the number of items in 
the scale [16]. Although the response distributions for 
the total score and Communication subscale were 
approximately Normal with no evidence of a ceiling 
effect, this was not the case for the Navigation subscale. 



There was a marked ceiling effect for this subscale, sug- 
gesting it may have limited usefulness as a stand-alone 
measure. Further development of this subscale, through 
inclusion and testing of additional items is warranted. 

'Care coordination' and 'continuity of care' are related 
but distinct concepts [21-23]. While 'care coordination' 
broadly addresses process issues relevant to streamlined 
and appropriate navigation of the health care system, 
'continuity of care' focuses more on consistency of 
information and clinical management between providers 
and over time, and on continuity within relationships 
[21-23]. As a result, measures of continuity of care have 
often focused on the issue of whether a patient saw the 
same doctor at each follow up visit [10,24,25]. Although 
seeing the same doctor is desirable in certain circum- 
stances, for example within a specific clinic, this aspect 
of care is less relevant for a broad assessment of the 
coordination of cancer care which is often multidisci- 
plinary in nature, involving consultations with a number 
of different health professionals where good communi- 
cation and exchange of information is paramount. Other 
existing questionnaires have focused on assessment of 
patients' experience of hospital discharge, in recognition 
that the risk of poor care coordination is particularly 
high at times of transition in care [11,26]. 

In the non-oncology setting, McGuiness and Sibthorpe 
took a broad approach to measuring health care coordi- 
nation, developing an instrument for older patients with 
chronic, complex medical conditions in the primary care 
setting [12]. Others have included a single or small 
number of relevant items pertinent to care coordination 
within questionnaires to assess perceptions of the qual- 
ity of care or satisfaction with cancer treatment gener- 
ally, however this approach limits the depth of 
information gathered specifically about care coordina- 
tion [27,28] In contrast, our instrument was designed to 
provide a more comprehensive assessment of cancer 
care coordination based on the issues identified by 



Table 4 Distribution of scores 





Communication 


Navigation 


Total score 


Possible range 


13-65 


7-35 


20-100 


Mean (sd) 


47.9 (8.1) 


30.5 (4.2) 


78.3 (10.7) 


Median (IQR) 


48 (43-53) 


31 (28-34) 


79 (72-85) 


'Floor' - % with lowest possible score 


04 


0 


0 


'Ceiling' - % with highest possible score 


1.1 


20.9 


0 
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patients and clinicians in our previous qualitative 
research. 

Overall, rates of item completion were high, suggest- 
ing that the questions were clear and acceptable to 
patients. Of note, the highest rates of missing data were 
for items asking about family and carer issues. The rea- 
son for this warrants further investigation as it could be 
that some patients do not have a carer, or do not iden- 
tify their family or friends as 'carers', or are unable to 
answer questions relating to the experience of their 
carers. Assessment of the experience of cancer care 
coordination from the perspective of carers warrants 
further research. 

A number of limitations to this study are acknowl- 
edged. Although our sampling strategy aimed to include 
a broad range of patients in Sample 1, there was a pre- 
ponderance of those with colorectal cancer and people 
from metropolitan centres. Furthermore, people with 
limited English skills were excluded from both samples 
and so the questionnaire may not be applicable to those 
from culturally and linguistically diverse communities. 
The methods used in this study were to reduce the 
number of questionnaire items to those with good psy- 
chometric properties and to provide a brief question- 
naire. It is possible that the resulting instrument omits 
important aspects of cancer care coordination and the 
development of additional items and subscales could 
improve the content validity of the instrument. Further- 
more, the responsiveness of this instrument to change 
needs to be tested in future studies. 

Conclusion 

In conclusion, the questionnaire developed in this study 
has been shown to be a psychometrically robust patient- 
report measure of cancer care coordination. Further stu- 
dies will help establish the usefulness of this measure in 
future needs assessment surveys and intervention 
studies. 

Additional material 



Additional file 1: Cancer Care Coordination Questionnaire for 
Patients. Copy of the 20-item questionnaire for patients. 
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